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(54) GENERATING DEVICE AND REPRODUCING DEVICE FOR PLAYING IMAGE 
INFORMATION 

(57)Abstract: 

PROBLEM TO BE SOLVED: To obtain the playing image information generating 
device which can play music and display image formation showing the state of the 
playing by generating playing formation on the music to be played and operation 
description information corresponding to the playing style and generating sequence 
formation including the operation description information and playing formation. 
SOLUTION: When sequence data by parts begin to be generatedthe playing 
formation is generated first (S21). Consequentlyplaying formation on the parts is 
generated. The playing styles of music of the composed parts are determined 

(522) . Operation components corresponding to the inputted playing formation and 
playing styles are selected and read out of an operation component data base 20 
to generate a basic operation waveform to be arranged and on the basis of the 
basic operation waveforma 3D image showing the state of the playing is generated 

(523) . Image editing processes such as arranging processes for operation 
waveform by the partsalterations of coordinates of respective positions or 
partsand the change of the position of a muting marker are performed (S25). 



CLAIMS 



[Claim(s)] 

[Claim 1]A performance picture information preparation device characterized by 
comprising the following for creating performance picture information for displaying 
a picture which shows a situation of a performance of this musical piece with a 
performance of a musical piece. 

It has an operation-parts database which stores operation parts which recorded a 
locus of operation by which every musical instrument or a player [ in / whole part 
/ the typical rendition ]and a musical instrument were subdividedA means to 
create behavioral description information by which information which specifies said 
operation parts corresponding to the performance information and rendition on a 
time schedule which makes a unit musical time bases which should be 
performedsuch as ** of a musical piece or a vibrant tunehas been arranged. 
This behavioral description information and said performance information. 

[Claim 2]Said performance picture information preparation device according to 
claim 1 wherein a means to create said behavioral description information is made 
as [ store / by using an edit result as operation parts / in said operation-parts 
database / it / it has the function to edit said specified operation partsand ]. 
[Claim 3]Said performance picture information preparation device according to 
claim 1 wherein said operation parts record a locus of operation for each part of 
said players bodyor each part of said musical instrument. 

[Claim 4]Said performance picture information preparation device according to 
claim 1 wherein said operation parts have a silence point marker in which a 
waveform showing a locus of said subdivided operation of operationa pronunciation 
point marker in which timing of pronunciation is shownand timing of silence are 
shown. 

[Claim 5]Said performance picture information preparation device according to 
claim 1 wherein setting out to each of time resolution of a waveform of operation in 
said operation parts is enabled. 

[Claim 6]On said time schedulehave a means to create scene information by which 
scene parts in said picturesuch as background information and light source 
informationhave been arrangedand said sequence informationSaid performance 
picture information preparation device according to claim 1 including said 
performance informationsaid behavioral description informationand said scene 
information. 

[Claim 7]Said performance picture information preparation device according to 
claim 6wherein information which directs to display information inputted from the 
outside as said scene part is included. 

[Claim 8]A performance picture information preparation device characterized by 
comprising the following for creating performance picture information for displaying 
a picture with a performance of a musical piece synchronizing with a performance 
of this musical piece. 

A means to create information by which information which directs to display 



picture information inputted from the outside on a time schedule which makes a 
unit musical time bases which should be performedsuch as ** of a musical piece 
or a vibrant tunehas been arranged. 

Created this information and performance information of said musical piece. 

[Claim 9]Said performance picture information preparation device according to 
claim 8wherein said information to direct includes information which controls a 
visual effect over information inputted from said outside. 

[Claim 10]It has a means to create behavioral description information by which 
information which specifies operation parts which recorded a locus of operation by 
which a player corresponding to the performance information and rendition and a 
musical instrument were subdivided on a time schedule which makes a unit 
musical time bases which should be performedsuch as ** of a musical piece or a 
vibrant tunehas been arrangedSaid performance picture information preparation 
device according to claim 8wherein said sequence information includes information 
by which said information to direct has been arrangedand said behavioral 
description information. 

[Claim 11]A performance picture-information reproducing device which is provided 
with the following and characterized by said picture preparing part creating said 
picture using said operation parts read from said operation-parts database based 
on said behavioral description information. 

Based on sequence information including behavioral description information for 
displaying a picture which shows a situation of a performance of a musical piece 
corresponding to performance information and this performance informationA 
musical sound generating part which is a performance picture-information 
reproducing device which displays a picture which shows a situation of the 
performance while generating musical tone corresponding to the performance 
information concernedand creates musical tone based on said performance 
information. 

A picture preparing part which creates a picture which shows a situation of a 
performance corresponding to said performance information based on said 
behavioral description information. 

An operation-parts database which stores operation parts which recorded a locus 
of operation by which a player and a musical instrument in the typical rendition 
were subdivided for every musical instrument or every part. 

[Claim 12]Said performance picture-information reproducing device according to 
claim 1 1 wherein said picture preparing part amends skeleton data in which a size 
of a player and a musical instrument which are contained in said operation parts is 
shown based on information which specifies a size of a display image contained in 
said sequence information and creates a display image. 

[Claim 13]Said performance picture-information reproducing device according to 
claim 1 1 characterized by comprising the following. 

An input means which inputs picture information from the device exterior. 



An image compositing means which combines a picture created by said picture 
preparing part and a picture from said input means. 

A means to give a visual effect to timing specified by a musical time basis of a 
musical piece to perform to a picture combined by a picture created by said 
picture preparing parta picture from said input meansor said synthesizing means 
based on said sequence information. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the performance picture 
information preparation device and playback equipment which can createedit and 
reproduce picture informationsuch as a picture which shows the situation of a 
performance of performance information and the musical piece concerned. 
[0002] 

[Description of the Prior Art]Conventionallythe application program called the 
multimedia authoring tool which sticks 2 in all dimensional images and a movie on 
the musical piece is known using general purpose computerssuch as a personal 
computer. 
[0003] 

[Problem(s) to be Solved by the Invention]In the application program mentioned 
aboveas a picture displayed with a musical pieceit is a still picture or the image 
data of two-dimensional (2D)and the picture of three-dimensional (3D) was not 
able to be displayed. When editing a display imageediting according to the 
performance of a musical piece was not easy. It was difficult to synchronize the 
picture to display and the musical piece to perform thoroughly. For exampleit was 
difficult to set a display image by the changewhen the playing tempo of a musical 
piece is changedor to give this effect synchronizing with a performancewhen giving 
a visual effect to a picture. 

[0004]Thenan object of this invention is to provide the performance picture 
information preparation device and playback equipment which can display the 
picture information which shows the situation of the performance with the 
performance of a musical piece. It aims at providing the performance picture 
information preparation device and playback equipment into which a picture can be 
edited according to a musical piece. Reproducing a picture synchronizing with the 
performance of a musical pieceor giving a visual effect aims at providing the 
performance picture information preparation device and playback equipment which 
can be performed easily. 
[0005] 

[Means for Solving the Problem]To achieve the above objectsa performance 
picture information preparation device of this inventionlt is a performance picture 



information preparation device for creating performance picture information for 
displaying a picture which shows a situation of a performance of this musical piece 
with a performance of a musical piecelt has an operation-parts database which 
stores operation parts which recorded a locus of operation by which every musical 
instrument or a player [ in / whole part / the typical rendition ]and a musical 
instrument were subdividedA means to create behavioral description information 
by which information which specifies said operation parts corresponding to the 
performance information and rendition on a time schedule which makes a unit 
musical time bases which should be performedsuch as ** of a musical piece or a 
vibrant tunehas been arrangedlt has a means to create sequence information 
including this behavioral description information and said performance information. 
[0006]A means to create said behavioral description information has the function 
to edit said specified operation partsand is made as [ store / in said operation- 
parts database / it / by using an edit result as operation parts ]. Said operation 
parts were having a locus of operation for each part of said player's bodyor each 
part of said musical instrument recorded. Said operation parts have a silence point 
marker in which a waveform showing a locus of subdivided operation of operationa 
pronunciation point marker in which timing of pronunciation is shownand timing of 
silence are shown further again. Setting out to each of time resolution of a 
waveform of operation in said operation parts is enabled further again. It shall have 
a means to create further again scene information by which scene parts in said 
picturesuch as background information and light source informationhave been 
arranged on said time scheduleandas for said sequence informationsaid 
performance informationsaid behavioral description informationand said scene 
information shall be included. Information which directs to display further again 
information inputted from the outside as said scene part is included. 
[0007]Further again other performance picture information preparation devices of 
this inventionlt is a performance picture information preparation device for 
creating performance picture information for displaying a picture with a 
performance of a musical piece synchronizing with a performance of this musical 
pieceA means to create information by which information which directs to display 
picture information inputted from the outside on a time schedule which makes a 
unit musical time bases which should be performedsuch as ** of a musical piece 
or a vibrant tunehas been arrangedlt has a means to create sequence information 
including created this information and performance information of said musical 
piece. Said information to direct includes further again information which controls 
a visual effect over information inputted from said outside. Further againlt has a 
means to create behavioral description information by which information which 
specifies operation parts which recorded a locus of operation by which a player 
corresponding to the performance information and rendition and a musical 
instrument were subdivided on a time schedule which makes a unit musical time 
bases which should be performedsuch as ** of a musical piece or a vibrant 
tunehas been arrangedSaid sequence information is having information by which 
said information to direct has been arrangedand said behavioral description 



information included. 

[0008]Further again a performance picture-information reproducing device of this 
inventionBased on sequence information including behavioral description 
information for displaying a picture which shows a situation of a performance of a 
musical piece corresponding to performance information and this performance 
informationA musical sound generating part which is a performance picture- 
information reproducing device which displays a picture which shows a situation of 
the performance while generating musical tone corresponding to the performance 
information concernedand creates musical tone based on said performance 
informationA picture preparing part which creates a picture which shows a 
situation of a performance corresponding to said performance information based 
on said behavioral description informationlt has an operation-parts database which 
stores operation parts which recorded a locus of operation by which a player and a 
musical instrument in the typical rendition were subdivided for every musical 
instrument or every partSaid picture preparing part creates said picture using said 
operation parts read from said operation-parts database based on said behavioral 
description information. Said picture preparing part amends skeleton data in which 
a size of a player and a musical instrument which are contained in said operation 
parts is shown based on information which specifies a size of a display image 
contained in said sequence informationand creates a display image further again. 
Furthermorean input means which inputs picture information from the device 
exteriorAn image compositing means which combines a picture created by said 
picture preparing part and a picture from said input meansBased on said sequence 
informationit has a means to give a visual effect to timing specified by a musical 
time basis of a musical piece to performto a picture combined by a picture created 
by said picture preparing parta picture from said input meansor said synthesizing 
means. 
[0009] 

[Embodiment of the Inventionj Drawing 1 is a block diagram showing an example of 
the composition of a device which operates as the performance picture 
information preparation device and playback equipment of this invention. A central 
processing unit (CPU) with which 1 controls operation of this whole device in this 
figureProgram storage which memorizes the control program with which 2 controls 
this deviceThe sequence data in which 3 is created using this device (sequence 
information)The operation-parts database which parts-ized the locus of the 
playing action in each renditionand stored itThe scene part database which stored 
scene partssuch as background information and light source informationWhile 
memorizing other various datathe memory storage which consists of ROMRAMetc. 
which are used as workspaceand 4 are operation switch groups which consist of 
various kinds of handlers provided in the keyboard (keyboard) and the navigational 
panel. 5 is a sound source part and generates the musical sound signal for two or 
more channels. This sound source part may be a thing of what kind of 
methodssuch as a waveform memory methodFM methoda physical model methoda 
harmonics composite systema formant composite systemand an analog 



synthesizer method of VCO+VCF+VCA. It is not restricted to the sound source 
circuit constituted using hardware for exclusive useand may be constituted by the 
program of the sound source circuit constituted using DSP and the micro 
programor CPU and software. The effect processing part for giving various kinds 
of effectssuch as a vibrato and RIBABUto the generated musical tone is also 
contained in this sound source part 5. 6 is a sound system for carrying out sound 
emission of the musical tone outputted from said sound source part 5. 
[0010]The image arithmetic unit (drawing engine) for 7 generating 3D image data 
or adding various kinds of visual effects and 8 are image display devices (graphic 
display) which display 3D picture generated by the image arithmetic unit 7the 
screen for edit mentioned lateretc. It may be made to make said CPU1 perform 
image processingwithout using said image arithmetic unit 7. It is a MIDI interface 
circuit for 9 to perform external storagessuch as a hard diska floppy diskCD- 
ROMMOand DVDand for 10 perform communication with external MIDI apparatus. 
A video interface circuit for 1 1 to display the picture which shows the situation of 
said performance to the monitor 12 connected outsideA picture and video input 
devicessuch as a television camera for 13 to input a picture or a video signal from 
the device exterior and VTRand 14 are the buses for performing the data 
communications between said each component. 

[0011]In this invention constituted in this wayit has the operation-parts database 
which stored the operation parts which recorded the locus of the operation by 
which the player and musical instrument in the typical rendition were subdivided 
for every musical instrument or every part. And the performance picture 
information preparation device of this invention reads the operation parts 
corresponding to performance information and the specified rendition from this 
operation-parts databaseEdit creation of the behavioral description information 
which shows the situation of a performance of the musical piece based on these 
operation parts is carried outand the sequence information which has said 
performance information and behavioral description information is outputted. The 
performance picture-information reproducing device of this invention generates 
the picture which reads operation parts corresponding from said operation-parts 
database based on the behavioral description information included in said 
sequence informationand correspondsSynchronizing with the performance of a 
musical piecethe repeat display of the picture inputted is carried out from the 
generated this picture and the outside. 

[0012][Operation-parts database] Said operation-parts database is explained first. 
This operation-parts database the locus of operation of the player and musical 
instrument in that typical rendition for every various musical instruments or every 
partincorporating as for examplemotion capture data — each part of a players 
bodyor each part (it is hereafter called a "part") of a musical instrument — each 
timewhile decomposing an operation locus into xyand z shaft orientationsit is this 
******** about the timing (for examplethe case of a drum RBI position) of 
pronunciation and silence — marking is carried out to data and it puts in a 
database. Herea waveform of operationand the subdivided data of a playing action 



are called "operation parts" for the operation locus decomposed into each shaft 
orientations of said xyz. 

[0013] Drawing 2 is a figure showing an example of the operation parts of a drum 
part. As known well to the typical rendition of a drum. As there are roll rendition (a 
single strokea double stroke)rim shot rendition (a closed rim shotan open rim 
shot)mallet renditionbrush renditionetc. and it is shown in this figureEach operation 
parts are constituted by the data of pronunciationa silence point markeretc. in 
which the timing of the waveform of operation by each rendition which shows the 
player at the time of the pattern performance and the operation locus of a musical 
instrument for every short performance patternpronunciationand silence is shown. 
In the drum part shown in this examplealthough the waveform of two or more 
musical instrumentssuch as cymbala snare drumand a bass drumof operation is 
used as one operation partsin the case of musical instrumentssuch as a piano and 
saxoperation parts are generated for every musical instrument. 
[0014]The preparation method of said operation parts is explained with reference 
to the flow chart of drawing 3 . Firstin Step S11a player's when the player's is 
performing by specific rendition by the specific musical instrument operationand 
operation of a musical instrument are acquired as motion capture data, (a) of 
drawing 4 is a figure for explaining that situationas shown in this figureit equips the 
important section (square and shown position) of a player's body with 3D 
digitizerhas the specific pattern subdivided by said specific rendition performedand 
records a motion of the body of the player at that time. The magnetic thing or the 
optical thing is known as a 3D digitizer, (b) of drawing 4 is a figure showing the 
example in the case of recording operation of a musical instrumentand shows the 
case where shake operation of cymbal is recorded in this case. In this caseit is 
square and three 3D digitizers attached to the shown position perform motion 
capturing. And in Step S12in the motion capture data which was carried out in this 
way and acquiredthe locus of the central point of each part is decomposed into a 
xyz coordinate systemand the waveform of operation which shows the movement 
state and position of each part is acquired. At this timetemporal data may also be 
recorded simultaneously. In the example of said cymbalthe waveform of said 
center position A of three points of operation is made into the waveform of a part 
of operationand three points are treated as what constitutes the same flat surface 
as A. The skeleton size information which shows the size of the target player or a 
musical instrument is also recorded. 

[0015]Nextit progresses to Step S13It is remembered as a marker (it is called a 
pronunciation point marker or a silence point marker) that the coordinates of the 
key-point part of the position (pronunciation point) from which pronunciation took 
placeand the position (silence point) of silenceand the lapsed time from the 
performance start can be distinguished. For examplewhen it is the performance of 
the phrase shown in (c) of drawing 4t hree positions shown in a figure serve as a 
pronunciation pointand each lapsed time tt'and t" are memorized so that 
distinction is possible. As long as this pronunciation point marker and the silence 
point marker can specify the timing of pronunciation and silence now within a set 



of said acquired data point (xyzt) of operationthey may be a thing of what kind of 
form. And it progresses to Step S14 and matching with the data acquired as 
mentioned above and the performed rendition is performed. At this timeparts 
which have a close relation mutuallysuch as each joint of a right arma shoulderan 
elbowa wristand a fingerare summarizedand they are stored as a set partfor 
example. The operator can rearrange this set part arbitrarilyor can decompose. 
[0016]Thusit puts in a database as data of form which can respond to change 
(change of the shape of a player and a musical instrumentor a size) of the position 
at the time of reproductionor change (change of tempo) of time. For 
exampleoperation until it finishes striking the operation parts about a drum from 
the operation which strikes each kit is stored as operation parts. For examplesince 
the stick was flung upcymbal strikes cymbaland the period until it flings up a stick 
again becomes operation parts. Herethe time of a stick hitting cymbal serves as a 
pronunciation point. Although it is not necessary to necessarily provide about a 
silence pointtiming which vibration of cymbal stopped can also be considered as a 
silence point. About a pianothe operation by the place which separates a keyboard 
from the place which is flipping the keyboard serves as operation parts to each 
finger (1 or two combination). Herethe timing to which the finger was equivalent to 
the keyboard serves as a pronunciation pointand the timing in which the finger 
carried out key-release serves as a silence point. And two or more variations are 
prepared according to the difference condition of a finger when 
flippingrespectivelyand the state of the remaining fingers. To the typical code 
using three or more fingerstwo or more variations are prepared for the state of the 
finger which is flipping it in finger usage or the difference condition of a finger. It 
may be made to also include the data of the movement speed for every 
partaccelerationetc. mentioned above in said operation parts besides each data of 
xyz coordinates and timeand skeleton size and pronunciationand a silence point 
marker. The data about the shape and textures of a player or a musical instrument 
is also recorded. 

[0017][Creation of sequence data] Nextcreation processing of said sequence data 
is explained. Drawing 5 is a flow chart which shows creation processing of 
sequence data. As shown in this figurecreation of sequence data is divided roughly 
into creation processing of the sequence data of all the parts which unify the 
sequence data of creation processing of the sequence data for every part and 
each created part. 

[0018](a) of drawing 5 is a flow chart of creation processing of the sequence data 
for every part. A start of creation processing of the sequence data for every part 
of this will perform creation processing of the performance information of Step 
S21 first. It is the same processing as composition of the musical piece to which 
this processing is performed in the usual sequence softwareandtherebythe 
performance information (MIDI data) of this part is created. 

[0019]Thenit progresses to Step S22 and the rendition of the musical piece of this 
part composed in said step S21 is determined. It can opt for this determination for 
every field selected on the score corresponding to said performance information. 



When this part is for examplea keyboard parta codefingering informationand 
informationincluding volume etc.are inputted. In the case of a drum partthe name 
of rendition is inputted. Since the operation parts matched from the operation- 
parts database 20 mentioned above by these information will be readlt directs to 
input information required in order to search a part operation waveform from said 
operation-parts database 20 according to the musical instrument or part which the 
operator chose. 

[0020]If the whole of this required information is readywill progress to Step 
S23and the operation parts corresponding to said inputted performance 
information and rendition will be chosen and read from said operation-parts 
database 20The basic motion waveform which is the target of edit is created3D 
picture which shows the situation of the performance based on the basic motion 
waveform is generatedand it displays on a screen. Firstthe player by which initial 
setting is beforehand carried out to this musical instrument or part is displayed 
with the musical instrument corresponding to the tone color information directed 
within the performance information of said musical piece. The operator can change 
into other players and musical instrumentswhen the player or musical instrument 
of a basic motion waveform is not pleasing (Step S24). Howeverabout the musical 
instrumenttop priority is given to the directions information in the musical piece 
concerned. For examplewhen directions information is grand pianoit is possible to 
make the graphics of grand piano into a grain tone from a black coloror to change 
transparentlybut it is preventing from being changed into the upright piano and 
other musical instruments from which a category differs. The choice of the player 
and musical instrument which can be changed is displayed on a screen each 
timeand it may be made to raise operativity at this time. 

[0021]Thusafter a player and a musical instrument are determinedan operator 
reproduces this basic motion waveform brieflyand when it is not pleasinghe will 
perform that editing processing. That isit progresses to Step S25 and image 
editing processingssuch as change of the coordinates (position) of the editing 
processing of a waveform of operation for every parteach partor a part and change 
of the position of a pronunciation silence markerare performed. Also when the 
player or musical instrument which is pleasing in said step S24 is not able to be 
chosenit progresses to this step S25and it is begun directly to read operation 
parts and they can be edited. The result edited by this image editing processing 
S25 is stored in said operation-parts database 20 as new operation partsand it is 
made as [ aim at / reuse of parts ]. 

[0022] Drawing 6 is a figure showing the example of the image editing screen 
displayed on said display 8when performing said step S23 - processing of 25. As 
shown in this figurein this image editing screen. Reproduction rewinding processing 
of a picture. The control window 30 which displays the frame position etc. of the 
image data used as the control button to perform or an editing objectEach part so 
that the image display window 40 which displays the 3D graphics picture which 
shows the situation of a performancethe player currently displayedand a musical 
instrument can be chosen per part with the layered structure. Four windows of the 



part part display information window 60 which displays the part of the part 
selection information window 50 to display and operation parts or the time 
schedule for every part are provided. 

[0023]Herethe button in which the control button 31 provided in said control 
window 30 returns to a head positionthe button in which 32 returns before 1 
vibrant tunethe button which 35 follows a reverse-direction-reproduction button 
and 34 to an earth switchand follow it to a forward direction reproduction button 
and in which 33 follows 36 to 1 vibrant-tune pointand 37 are buttons which go to 
end position. The operator can control reproduction of the 3D picture information 
concerned in said image display window 40 by operating these buttons. 
[0024]In said step S23a player (player) and a musical instrument with the selected 
operator are displayed on said image display window 40. At this timethe skeleton 
size datashapeand texture data which were mentioned above as initial information 
are used. The operator can perform change setting out of that position by 
choosing the part of the player and musical instrument which are displayed on this 
image display window 40and shifting that position. Also by choosing the part 
display information in said part selection information window 50It is supposed that 
it is possible to perform said selectionand it becomes shade and can be 
considered as a selectable displaying condition in said image display window 40 
about the part where it is difficult to choose by choosing by said part selection 
information from the picture currently displayed. Simultaneously with setting out of 
this positionchange processing of the coordinates of a waveform of operation is 
also performed automatically. 

[0025]Each part which constitutes these operation parts is displayed on said part 
selection information window 50 with that layered structureand the operator is 
made as [ choose / the part currently displayed on this window ]. In the example 
to illustrateeach part of the drum player and the drum kit is displayed 
hierarchicalrespectively. 

[0026]The field 61 which displays the performance information of the partand the 
field 62 which displays the time schedule of each part are established in said part 
display information window 60. If the waveform of the part for one music of 
operation is createdoperation parts will display ** for the time schedule for every 
set-ized part as unit timefor example. The dashed line of ****** shows ** and the 
solid line shows the pause of a vibrant tune. The waveform of each part of 
operation is arranged considering ** as a unit at the position for which the 
waveform of operation in the time schedule of the part is used. In drawing 6 63 
shows the time schedule of a part called a drummerand 64 and 65 show the 
waveform of the part about a musical instrument of operationrespectively. Each 
waveform of operation is displayed in the rectangleand the position is shown by 
the rightward triangle about what includes a pronunciation marker in it. In the 
illustrated exampleit turns out that the stick is contained in the part 63 and the 
position of a pronunciation marker is displayed. Thereforethe performance 
information of the already inputted musical piece can be displayed according to a 
partand the timing of pronunciation and silence can be checked. If the 



pronunciation marker of a waveform of operation and a silence marker are set by 
the timing of this pronunciation and silencegenerating of musical tonesilencethe 
pronunciation of the performance in a pictureand silence operation can be 
synchronized. 

[0027]The reproduction rate of a waveform of operation of the part can be 
changed by choosing the rectangle which shows the waveform of each part of 
operationand changing the length. For examplea reproduction rate can be lessened 
by lengthening the length. A reproduction rate is automatically changedalso when 
the tempo of the musical piece concerned is changed. The straight line made 
square [ both ends ] is displayed on the inside of the rectangle of each operation 
waveformand this straight line expresses the portion (portion actually reproduced) 
used at the time of reproduction of the picture of those operation waveforms with 
it. By changing the length and the position of this straight linethe portion used for 
reproduction of those operation waveforms can be chosen. When the length of 
said rectangle is changedthe length of this straight line maintains that relative 
relationand is changed. A change of such length will be made considering musical 
time basessuch as ** or a vibrant tuneas a unit. Each part included in the part is 
made selectable in said image display window 40 by choosing said rectangle of a 
waveform of operation further again. Therebychanging a waveform of operation 
into other variation candidates for every part and edit of the waveform of 
operation itself are attained. 

[0028]An operating frame number can also be changed for every part further again 
by making into a maximum the frame number by which motion capture was carried 
out. A frame number is made to increase about the part which operates finely by 
thisand the frame number of the part which seldom operates can be decreased 
and can make the load of an operation reduce. It explains taking the case of 
composition of the waveform of each part of the drum part shown in drawing 7 of 
operation. As shown in this figurefive kinds of waveformsoperation of human being 
except the (a) flngeroperation of the (b) fingeroperation of (c) stock musical 
instrument (drum stick)operation of (d) deferment musical instrumentand operation 
of the (e) flexible regionof operation are used for a drum part. At this 
timeoperation of human being of (a) and operation of the stick of (c) are used as 
the set partand the frame number is enlarged. Since operation of the (b) 
fingeroperation of (d) deferment musical instrumentand (e) flexible region operation 
do not need a high resolutionthe frame number is decreased so that it may 
illustrate. 

[0029]The direct edition of said waveform of operation can also be carried out 
further again. Drawing 8 is a figure showing the situation in the case of carrying 
out the direct edition of this waveform of operation. Herethe case where the 
waveform of a part "head (HeadA)" of operation is edited is shown. If the part 
HeadA to edit is chosen in said part selection information window 50 or said image 
display window 40in order to influence the part which the low order hierarchy has 
by the edit result of HeadAIn said part selection information window 50the part (59 
in a figure) in HeadA and its low rank is expressed as a different gestalt (for 



examplewhite) from other parts. In 3D picture currently displayed on said image 
display window 40 corresponding to thisAs shown in [ B ] a figurethe field of the 
part (namelypart which has a relation restrained by the selected part) which the 
hierarchy of the this chosen part and its low rank has is displayed by a different 
display style (for examplethe luminosity of the field becomes high) from other 
parts. If the part which the operator chose is specified by double click etc.the 
waveform display window 70 of operation shown in drawing 8 will be displayedand 
the waveform of the part of operation will be displayed. The waveform of operation 
shown in this figure is a thing of HeadAand the movement angle to the initial 
coordinate position of HeadA of the direction vector which goes to HeadA from 
the part Sternum of that low rank (sternum) is shown. In the figurea horizontal axis 
shows a frame numberthe vertical axis shows the angleand three graphs showing 
the angle from [ from a top ] a x axisthe y-axisand the z-axis are shown. 
[0030]If an operator makes the position of a part to edit in said image display 
window 40 changein connection with thissaid waveform of operation will be 
changed automatically. If the waveform of operation currently displayed on said 
waveform display window 70 of operation is edited converselythe 3D picture 
currently displayed on the image display window 40 according to this will operate. 
Thusit is also possible to carry out the direct edition of said waveform of operation. 
Edit of this operation waveform is effective when it compounds two or more sorts 
of operationsfor exampleand making that knot smooth. 

[0031]Thusafter editing a waveform of operationa name can be added to the 
waveform of the this edited part of operationand it can add to said waveform 
database 20 of operation. The waveform of a part of operation is possible also for 
carrying out continuation composition of the two or more kindsor making it 
decomposeand can also add these to a database newly. Nowdetermination of the 
file of operation which does in this way and shows the situation of a performance 
of this part will memorize automatically that waveform file name of operationa 
pointer (instrument informationrendition)etc. corresponding to performance 
information. 

[0032]Thusafter the sequence data according to each part is createdprocessing 
which unifies the sequence data of all the parts shown in (b) of drawing 5 and 
adjusts the whole is performed. This processing is performed using the screen for 
edit of all the parts shown in drawing 9 . In Step S31a start of this processing will 
perform attachment processing of the data of each part first. In this 
processingeach part is specifiedrespectively and both musical tone and a picture 
are arranged on the time schedule. In drawing 9 80 is a window which displays the 
time schedule of all the partsand the field 81 which arranges the scene information 
about a stageand the field 82 which displays the data of each part are formed. 
Here** and a solid line express the vibrant tune and a vertical dotted line will be 
arranged on the absolute vibrant tune within the one musical piece concernedand 
** by arranging each part information here. In the edit display according to said 
drawing 6 and part of drawing 8 relative ** from the starting point of the part and a 
vibrant tune are shown. 



[0033]Nextit progresses to Step S32 and editing processing of scene information 
is performed. As mentioned abovethe scene part database is stored in said 
memory storage 3and various kinds of scene parts for forming pictures about a 
performance stagesuch as background informationinformation on a light sourceand 
information on a camera (viewpoint)are stored in this scene part database. Thenin 
this step S32scene partssuch as background informationinformation on a camera 
(viewpoint)and information on a light sourceare chosen from said scene part 
databaseand it sticks along with ** of said field 81 and a vibrant tune. In the 
example shown in drawing 9t hree camera information of G0-G2G3two background 
information of G4and the light source information of G5 are stuck on the time 
schedulerespectively. These scene parts are described by VRML (Virtual Reality 
Modeling Language). 

[0034]The movie file created by the on-the-spot photo image inputted from said 
picture and video input devices 13such as a television camera and VTRor 
computer is incorporatedand the control information for using it as background 
information etc. is also stored as scene parts. This input on-the-spot photo image 
can be displayed as a backgroundor can be displayed on viewing areas arbitrary as 
a part of background. For exampleas for what (chromakey synthesis) a photograph 
is taken or a CG image and an on-the-spot photo image are compounded also for 
with the transmissivity arbitrary in starting only colors other than the back and 
sticking on a picture as a background which generates computer graphics 
converselythe back can do single colorssuch as blue. If this combines the picture 
generated from said waveform of operation against the background of the on-the- 
spot photo image or human being of other musical instrument parts also 
incorporates as scenery (background)the scene which is on a display and is 
performed together with CG player can be displayed. It is also possible to make it 
display only an on-the-spot photo imageto insert an on-the-spot photo image and 
to heighten a playing effect into an intro or an interludeby this. Since the control 
information for incorporating this on-the-spot photo image is also scene partsit is 
stuck like other scene parts on the time schedule in alignment with musical time 
basessuch as ** and a vibrant tune. 

[0035]Effect processing to a picture can also be used as said scene part as said 
scene part. This scene part can stick on said time scheduleand a ******** 
caseThe change of the viewpoint of the picture pictured on the monitor which 
combined said on-the-spot photo image and said generated pictureand a 
background changeandsometimesfade-inmaking fade-outwipeetc. processmaking 
some display images into a mosaicor making a color tone into sepia **** — etc. - 
- it becomes possible to give various visual effects. The scene parts about such 
effect processings can be registered into said scene part database as a 
functionand can add arbitrary processings. 

[0036]About this scene part as well as the case of the part mentioned aboveit can 
choose on the time schedule of said field 81 and can edit changing that length etc. 
It is also possible for each detailed information to be displayed and to perform the 
edit by choosing the stuck scene information. Thusthe completed music can 



appoint a field and can perform reproductionrewindinga rapid traverseetc. A track 
name etc. can be given and saved. 

[0037]With reference to drawing 10t he case where the information on such two or 
more parts is compounded is explained. As shown in (a) of drawing 10 a global 
stage coordinate system is set up and the coordinate system of the things unit 
which has the musical instrument or skeletal structure for every personal 
coordinate system for every player and player in this coordinate system is set up. 
Classifying said musical instrument and things into a rest frame and a braking 
systemthe thing of a braking system makes a flexible region the starting point of 
the coordinate system. Therebya coordinate changei.e.repositioningcan be freely 
performed in a global coordinate system. For examplein the case of the keyboard 
of a pianothe coordinate system which makes the starting point the portion which 
moves for every keyboard is set up and definedand if one kind of waveform of the 
keyboard of operation is preparedthe appropriation of data of it will be attained by 
[ which apply ] moving the home position for every keyboard. What is necessary is 
just to hold early operation about the musical instrument and things. [ of a rest 
frame ] Among thingsa musical instrumentetc.about the thing of the stock of 
playerssuch as a drum stick and a violina coordinate system local as the starting 
point is defined for a flexible region in the personal coordinate system for every 
playerand operation of operation parts is reproduced on this local coordinate 
system. When the player of each part and the size of a musical instrument are 
changeda ratio is calculated for every part and it amends in the size after changing 
the skeleton size in said operation parts. 

[0038]As shown in (b) of drawing 10t he sound which the player makesand 
operationi.e.performance information and performance informationtake a 
synchronizationand the package of each player of operation is package-ized. And 
the head position is doubled and the package of each player of operation is 
arranged. Thusthe behavioral description information on each player and a musical 
instrument is unifiedand the sequence information containing all the players is 
created. 

[0039][Composition of a sequence file] Drawing 1 1 is a figure showing the 
composition of a sequence file including the sequence information which did in this 
way and was created. As shown in this figurethe sequence file comprises three 
filesO) scene file(2) personal initial information fileand a (3) personal performance 
description file. And the information concerning [ the scene file of (1) ] this whole 
sequence file is includedScene informationsuch as reproduction speeda viewpoint 
and a viewa light sourcebackground informationand information that directs 
incorporation of the picture information from said outsidethe player of each part 
and the coordinate system position of a musical instrumentand size information 
are included. It may be made to hold scene informationsuch as a viewpoint and a 
viewa light sourceand background informationhere as pointer information to the 
scene parts concerned in said scene part database. The personal initial 
information file of (2) includes the information which determines the skeleton data 
of each player and a musical instrument and shapeand textures for every part. The 



personal performance description file of (3) includes the performance information 
file and behavioral description file (behavioral description information) of the part 
for every part. Herethe performance information file is made into MIDI data (SMF) 
formand the data point of a player and a musical instrument (tool) of operation by 
which the behavioral description file is included in said operation-parts database 
20and the pointer information to said pronunciation and a silence point marker are 
included. 

[0040]Thussince he is trying for the sequence file of this invention to hold the 
information for accessing not to the picture information itself but to an operation- 
parts database as a behavioral description fileit can make size of a sequence file 
small and has become a good thing of portability. 

[0041][Regeneration of sequence information] Nextwith reference to the flow 
chart of drawing 12 the processing which carries out the repeat display of a 
performance and picture of the musical piece concerned from the sequence data 
created by carrying out such is explained. In the sequence data creation 
processing mentioned abovewhen a reproduction button is operatedsame 
processing is performed. Determination of the music which an operator is going to 
perform will choose the sequence file (regenerative data) of the musical piece 
concerned from the music database 21 etc. with which the sequence file 
corresponding to the musical piece created as mentioned above is stored. In Step 
S41the sequence file (regenerative data) of this selected music is read specified 
length everyand processing of Steps S42 and S46 is performed based on this data. 
[0042] Step S46 is the automatic playing process known conventionally and the 
same processingand creates pronunciation eventssuch as a key ON event and a 
control changeand a sound-source-control parameter based on the performance 
information included in the read regenerative data. Thusit is inputted into said 
sound source part Scorresponding musical tone is generated (Step S47)and the 
created sound-source-control parameter is outputted from said sound system 6. 
[0043]On the other handin Step S42the data point of each part of operation 
corresponding from said operation-parts database 20 is read based on the 
behavioral description file in said read regenerative data (sequence file). And the 
compensation process of the coordinates position of a waveform of operation of 
each part followed and read to Step S43 is performed. When selection of a display 
part or the change of a view position is performed by the operator at this timethe 
compensation process of the coordinates position according to it is performed. If 
specifically necessary about the waveform of each part of operation so that the 
reproduction rate of the part may be suitedoperation data will be created by 
performing interpolation processingand it will amend by spline interpolation etc. 
about a bond portion of operation further. And corresponding to the position of 
each partetc.compensation processessuch as transmission of a coordinates 
positionrotationand expansionare performed. The scene parts specified in said 
scene file are read from the scene part database 22and scene information (stage 
information) is also generated. Thusin this step S23a model position is determined 
and animation is determined that it will correspond to it. 



[0044]To each operation parts stored in the operation-parts database 20 at this 
time as mentioned above. They are included by not only the coordinate data in 
alignment with a time-axis but pronunciation and a silence point markerand with 
this pronunciation point marker and a silence point marker. The time or speed from 
the coordinates of each pronunciation point and a silence point and the 
reproduction start of the operation waveform to pronunciation and a silence point 
is acquirable. Thereforehe is trying to take the synchronization with the image to 
generate and the musical tone to generate based on this pronunciation and a 
silence point marker. 

[0045]Namelywhen the tempo performed to tempo (reference tempo) when said 
operation parts are created is changed into k times as much tempo, operating 
waveform read-out of operation on a curtailed schedule so that it may reach from 
said reproduction start of a waveform of operation to a pronunciation point in 1/k 
time as much time (one the speed [ If it is speed ] of k times of this)and the 
reproduction interval of the operation waveform may become short or long **** - 
- multiple times — what is necessary is just to control like reading the same 
active position When transit time or movement speed is prepared for every 
coordinates (i.e.when time until it moves to the following coordinates from 
coordinates with each partor the information on speed is included in operation 
parts)What is necessary is to change k timesrespectively at the time of 1/k time 
and speed (amendment)and just to controlwhen it is time. 

[0046]By the wayit may become an unnatural picture only by controlling a time- 
axis simply as mentioned above about all the operations. For exampleif tempo is 
made into a halfon the wholethe working speed of a picture will turn into a half 
speedand in a performance of a drum etc.it will be a picture which is struck quietly 
and will be visible like the performance which stopped volume. What is necessary 
is to enable it to recognize the part about pronunciation operation in operation to 
a pronunciation point (from a pronunciation operation start point to the silence 
operating point)after starting operationand just to make it not change the working 
speed from a pronunciation operation start point to the silence operating pointin 
order to avoid thiseven if it changes tempo. 

[0047]It may be made to perform ornamentation of a waveform of operation using 
sound-source-control parameters generated in said sound-source-control 
parameter generation step S46such as an envelope and a velocity. When the 
operation parts concerned are shake operation of cymbalfor examplebased on 
musical-sound-control parameterssuch as a velocity or track volumeWhen a 
velocity or track volume is largethe shake of cymbal is enlargedand a waveform of 
operation is created so that a shake smallif conversely small. Therebythe situation 
of a natural performance can be displayed. 

[0048]Nextit progresses to Step S44 and image generation processing (rendering) 
is performed using the information determined by said step S43. That isconversion 
to signals of a scene is performed based on said scene information and a 
waveform of operation. Based on said scene information or a waveform of 
operationnamelythe peak of each objectThe three-dimensional coordinates 



calculation about a normalilluminationcoloringtexture coordinate calculationpolygon 
formationprojection coordinates calculationview clippingbit map creationa hidden 
surfacetransmission processingetc. are processed3D animation picture is 
generatedand it outputs to a frame buffer. When the sizes of the player and 
musical instrument which are actually displayed as the size of the player and 
musical instrument which were created beforehand at this time differa ratio is 
calculated for every partsaid coordinate information is amendedand the difference 
in skeleton size is amended. 

[0049]When the control signal which captures the image from said picture and 
video input device 13 as scene information is stuck as mentioned aboveTo the 
timing specified by this scene informationan on-the-spot photo image etc. are 
inputted from said picture and video input device 13and if neededit compounds 
with the picture generated by image generation processingand outputs. When the 
effect processing to a picture is included in scene informationin Step S45visual 
effect processing is performed to the picture created in said step S44. It can 
express as the picture which synchronized the state of the performance with the 
performance as mentioned above synchronizing with composition data. This output 
can also be saved as movie data. 

[0050]Were matched so that the candidate of operation parts might be searched 
and outputtedwhen the operation-parts database 20 inputted a musical piece and 
rendition abovebut. When the new music notation item which was matched and 
was used as the operation parts in a database is defined and a musical score is 
created using the notation rule and a signa picture may also be made to be 
illustrated automatically. 
[0051] 

[Effect of the Invention]As explained aboveaccording to this inventionit becomes 
possible to display the picture which shows the situation of the performance 
synchronizing with the performance of a musical pieceand the natural picture 
which synchronized with the performance of musical tone can be displayed. The 
picture which shows the situation of a performance of a musical piece can be 
edited easily. Displaying the on-the-spot photo picture inputted from the outside 
synchronizing with a performance and the effect which synchronized with the 
performance to the picture can be addedand a playing effect can be heightened. 
Since operation parts are put in a database further againwhile being able to use 
operation parts in common to two or more rendition and musical piecesrequired 
parts can be added to a database. Thereforeit becomes possible to generate a 
picture efficiently. Since it has the pronunciation point marker and the silence 
point marker with the waveform of operation as operation partsit becomes 
possible to use common operation parts to change of tempoetc.and it becomes 
possible to make size of a database small further again. 
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[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the example of composition of the 1 
embodiment of the performance picture information preparation device of this 
inventionand playback equipment. 

[Drawing 2] It is a figure for explaining an operation-parts database. 
[Drawing 3] It is a flow chart for explaining creation of operation parts. 
[Drawing 4] It is a figure for explaining creation of operation parts. 
[Drawing 5] It is a flow chart of sequence information creation processing. 
[Drawing 6] It is a figure showing an example of the image editing screen for every 
part. 

[Drawing 7] It is a figure for explaining composition of two or more operation parts. 
[Drawing 8] It is a figure showing an example of the edit display of operation parts. 
[Drawing 9] It is a figure showing an example of the image editing screen of all the 
parts. 

[Drawing 10] It is a figure for explaining composition of the information on two or 
more parts. 

[Drawing 1 1] It is a figure showing the composition of a sequence file. 
[Drawing 12] It is a flow chart of sequence information regeneration. 
[Description of Notations] 

1 CPU2 program storageand 3 Memory storage4 operation switch groupsFive 
sound source parts and 6 A sound system and 7 An image arithmetic unit and 8 
Image display deviceNine external storages'! 0 MIDI interface circuitsand 1 1 [ An 
operation-parts database 21 music databasesand 22 / Scene part database ] A 
video interface circuit and 12 A monitor and 13 A picture and a video input 
deviceand 20 



